The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Security gateways are used by a variety of organizations to protect their networks, internal users, and assets. A typical gateway consists of various scanning modules that are placed in a threat detection pipeline to filter out undesirable network data. An example of a threat detection pipeline includes a spam detection scanning module and an antivirus scanning module placed in a sequence.
The sequence of a threat detection pipeline is determined by system designers based on their knowledge of the network or domain that is protected. Once a system designer determines a sequence of scanners, the system is static. However, the threat landscape is continually evolving with new vulnerabilities being found in different software systems. Cyber criminals tend to gravitate towards these new vulnerabilities because they have a higher yield compared to old vulnerabilities. Thus, even if a module in a threat detection pipeline is able to identify a new threat exploiting these vulnerabilities, the security gateway must still account for large surges in traffic that attempt to capitalize on the new vulnerabilities.
One approach to handling surges of attacks is to update a module based on a new threat. For example, an email security appliance could add a new definition to an existing module. However, security devices cannot afford to stop looking for older threats, because internal systems may not have been upgraded against those vulnerabilities. So, even when new threats are being found, security devices continue to process payloads against a growing, static order of scanning modules. This approach is sub-optimal when a certain type of attack is placed on the system that is easily filtered out by one scanning module, but must go through many other scanning modules first. A significant number of cycles may be wasted on processing multiple data units before they are blocked. The results being that the gateway is overworked without much benefit.
Another approach to handling attacks is to define multiple threat detection pipelines called policies, where each policy includes a group of scanning modules. The benefit here is that some scanning modules may be bypassed in the security threat pipeline according to the rule set of the policy. For example, using an email security appliance (ESA), a policy classifier may look at the sender of an email and determine that such a sender is unlikely to mail spam. Thus, the gateway chooses a threat policy that does not include a spam filter. A setback of this approach is the requirement of knowledge regarding the application level data before going through the pipeline process. In addition, certain attacks will still be processed in a sub-optimal manner. For example, if an email address is known to forward both useful files and files containing spam, viruses, and malware, the ESA must continually send all data to a policy that has those three scanning modules using a predefined order. If there is a surge in traffic that includes a new threat that is detectable by the third module in the scanning order, the ESA continues to process the mail through the first two modules as well. For example, a current surge in mail may include spyware but not include detectable spam or viruses. The first two scanners use up “valuable” processing cycles scanning for spam and viruses, even though the mail is eventually discarded by a third anti-spyware scanning module.